Unsupervised Deep Learning: A Short Review

Deep neural networks with several layers have during the last years become a highly successful and popular research topic in machine learning due to their excellent performance in many benchmark problems and applications. A key idea in deep learning is to not only learn the nonlinear mapping between the inputs and outputs, but also the underlying structure of the data (input) vectors. In this chapter, we first consider problems with training deep networks using backpropagation type algorithms. After this, we consider various structures used in deep learning, including restricted Boltzmann machines, deep belief networks, deep Boltzmann machines, and nonlinear autoencoders. In the later part of this chapter we discuss in more detail the recently developed neural autoregressive distribution estimator (NADE) and its variants.